Evidence for Test Validation: A Guide for Practitioners.

BACKGROUND: Validity is a core topic in educational and psychological assessment. Although there are many available resources describing the concept of validity, sources of validity evidence, and suggestions about how to obtain validity evidence; there is little guidance providing specific instructions for planning and carrying out validation studies. METHOD: In this paper we describe (a) the fundamental principles underlying test validity, (b) the process of validation, and (c) practical guidance for practitioners to plan and carry out sufficient validity research to support the use of a test for its intended purposes. RESULTS: We first define validity, describe sources of validity evidence, and provide examples where each of these sources are addressed. Then, we describe a validation agenda describing steps and tasks for planning and developing validation studies. CONCLUSIONS: Finally, we discuss the importance of addressing validation studies from a comprehensive approach.

Evidence for Test Validation: A Guide for Practitioners / Evidencias Sobre la Validación de los Tests: una Guía Práctica

Background: Validity is a core topic in educational and psychological assessment. Although there are many available resources describing the concept of validity, sources of validity evidence, and suggestions about how to obtain validity evidence; there is little guidance providing specific instructions for planning and carrying out validation studies. Method: In this paper we describe (a) the fundamental principles underlying test validity, (b) the process of validation, and (c) practical guidance for practitioners to plan and carry out sufficient validity research to support the use of a test for its intended purposes. Results: We first define validity, describe sources of validity evidence, and provide examples where each of these sources are addressed. Then, we describe a validation agenda describing steps and tasks for planning and developing validation studies. Conclusions: Finally, we discuss the importance of addressing validation studies from a comprehensive approach.(AU)

Antecedentes: La validez es un tema central en la evaluación psicológica y educativa. A pesar de que la literature disponible recoge numerosos recursos en los que se describe el concepto de validez, las fuentes de evidencia y se aportan sugerencias sobre cómo obtener evidencias de validez, apenas existen guías que proporcionen instrucciones específicas para planificar y desarrollar estudios de validación. Método: El presente artículo describe (a) los principios fundamentales en los que se sustenta la validez de los test, (b) el proceso de validación, y (c) una guía práctica para planificar y recoger evidencias de validez que apoyen el uso de un test para alcanzar el objetivo previsto. Resultados: En primer lugar, se describe el concepto de validez y las fuentes de evidencia, aportando ejemplos específicos donde se abordan cada una de ellas. A continuación, se describe una agenda de validación en la que se enumeran los pasos y tareas necesarios para planificar y completar un estudio de validación. Conclusiones: Finalmente, se discute la relevancia de adoptar una aproximación comprehensiva al abordar estudios de validación.(AU)

Student Assessment Opt Out and the Impact on Value-Added Measures of Teacher Quality.

Student assessment nonparticipation (or opt out) has increased substantially in K-12 schools in states across the country. This increase in opt out has the potential to impact achievement and growth (or value-added) measures used for educator and institutional accountability. In this simulation study, we investigated the extent to which value-added measures of teacher quality are affected as a result of varying degrees of opt out, as well as a result of various types of nonrandom opt out. Results show that the magnitude of opt out and choice of classification scheme has a greater impact on value-added estimates than the type of opt-out patterns simulated in this study. Specifically, root mean square differences in value-added estimates increased as magnitude of opt out increased. In addition, teacher effectiveness classification agreement decreased as opt out magnitude increased. One type of opt out, where the highest achieving students in the highest achieving classrooms opted out, had the largest impact on stability than the other types of opt outs.

Differential ítem functioning: Beyond validity evidence base don internal structure / Funcionamiento Diferencial del Item: más allá de las evidencias de validez basadas en la estructura interna

Background: In the latest release of the Standards for Educational and Psychological Testing, Differential Item Functioning (DIF) is considered as validity evidence based on internal structure. However, there are no indications of how to design a DIF study as a validation study. In this paper, we propose relating DIF to all sources of validity evidence, and provide a general conceptual framework for transforming "typical" DIF studies into validation studies. Method: We perform a comprehensive review of the literature and make theoretical and practical proposals. Results: The article provides arguments in favour of addressing DIF detection and interpretation as validation studies, and suggestions for conducting DIF validation studies. Discussion: The combination of quantitative and qualitative data within a mixed methods research perspective, along with planning DIF studies as validation studies, can help improve the validity of test score interpretations (AU)

Antecedentes: en la última edición de los Standards for Educational and Psychological Testing, el Funcionamiento Diferencial del Item (DIF) es considerado como una evidencia de validez basada en la estructura interna. Sin embargo, no hay indicaciones claras sobre cómo diseñar un estudio de DIF como un estudio de validación. Proponemos relacionar el DIF con todas las fuentes de evidencias de validez y un esquema conceptual para transformar los estudios "típicos" de DIF en estudios de validación. Método: se lleva a cabo una extensa revisión de la literatura y realizan propuestas teóricas y prácticas. Resultados: el artículo aporta argumentos a favor de abordar la detección e interpretación del DIF como estudios de validación y recomendaciones para realizar estudios de validación sobre el DIF. Discusión: la combinación de resultados cuantitativos y cualitativos en un marco de investigación mixta, junto con el diseño de los estudios de DIF como estudios de validación, puede mejorar la validez de las interpretaciones de las puntuaciones en los tests (AU)

Differential Item Functioning: Beyond validity evidence based on internal structure.

BACKGROUND: In the latest release of the Standards for Educational and Psychological Testing, Differential Item Functioning (DIF) is considered as validity evidence based on internal structure. However, there are no indications of how to design a DIF study as a validation study. In this paper, we propose relating DIF to all sources of validity evidence, and provide a general conceptual framework for transforming “typical” DIF studies into validation studies. METHOD: We perform a comprehensive review of the literature and make theoretical and practical proposals. RESULTS: The article provides arguments in favour of addressing DIF detection and interpretation as validation studies, and suggestions for conducting DIF validation studies. DISCUSSION: The combination of quantitative and qualitative data within a mixed methods research perspective, along with planning DIF studies as validation studies, can help improve the validity of test score interpretations.

Using Bilingual Examinees to Evaluate the Comparability of TestStructure across Different Language Versions of a Mathematics Exam

Malay- and English-language versions of a mathematics exam were analyzed for structural equivalence by administering both versions to a group of Malay-English bilingual students. The analysis and comparison of test structure was determined using both DIMTEST and weighted multidimensional scaling. The assessment was found to be unidimensional and to possess similar structure across the two language versions. Implications of this study suggest bilingual examinees can be used to evaluate the invariance of test structure across translated test forms. Future research should explore situations where bilingual examinees can be used to link different language versions of assessments for monolingual populations.

Se analizó la equivalencia estructural entre las versiones de un examen de matemáticas en lengua malaya e inglesa mediante la administración de ambas versiones a un grupo de estudiantes bilingües en ambas lenguas. El análisis y comparación de la estructura del test fue realizada utilizando DIMTEST y escalamiento multidimensional ponderado. Se encontró que la evaluación es unidimensional y posee una estructura similar en las dos versiones. Las conclusiones de este estudio sugieren que se pueden utilizar personas bilingües para evaluar la invarianza de la estructura del test utilizando formas traducidas de un test. Las investigaciones futuras deberían explorar situaciones donde se puedan utilizar personas bilingües para conectar distintos idiomas en las evaluaciones de las poblaciones monolingües.

Validating assessments: introduction to the special section / Evaluaciones de validez: introducción a la Sección Especial

BACKGROUND: Validation is the process of providing evidence that tests and questionnaires are adequately and appropriately fulfilling the purposes for which they are developed. In this special issue, experts from several countries describe specific approaches to test validation and provide examples of their approach. These approaches and examples illustrate the validation framework implied by the Standards for Educational and Psychological Testing. METHOD: We describe the Standards' approach for building a validity argument based on validity evidence based on test content, response processes, internal structure, relations to other variables, and testing consequences. RESULTS: The five articles provide comprehensive examples of gathering data regarding these five sources of evidence and how they contribute to the validation of the use of test scores for particular purposes. CONCLUSIONS: These five articles provide concrete examples of how the five sources of validity evidence suggested by the Standards can be used to develop a sound validity argument to support the use of a test for its intended purposes

ANTECEDENTES: la validación es el proceso de aportar evidencias de que las evaluaciones mediante tests y cuestionarios cumplen adecuada y apropiadamente los objetivos para los que se elaboran. En este número especial expertos de varios países describen enfoques específicos para la validación y aportan ejemplos. Estos enfoques y ejemplos ilustran el marco de validación implicado por los Standards for Educational and Psychological Testing. MÉTODO: describimos la aproximación de los Standards para elaborar un argumento de validez a partir de evidencias de validez basadas en el contenido del test, los procesos de respuesta, la estructura interna, las relaciones con otras variables y las consecuencias del uso del test. RESULTADOS: los cinco artículos aportan ejemplos comprehensivos de obtención de datos en relación con las cinco fuentes de evidencia, y de cómo contribuyen a la validación del uso de las puntuaciones en el test para objetivos específicos. CONCLUSIONES: los cinco artículos aportan ejemplos concretos de cómo las cinco fuentes de evidencias de validez sugeridas por los Standards pueden utilizarse para elaborar un sólido argumento de validez que apoye el uso de test para sus objetivos previstos

Validity evidence based on test content / Evidencia de validez basada en el contenido del test

BACKGROUND: Validity evidence based on test content is one of the five forms of validity evidence stipulated in the Standards for Educational and Psychological Testing developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. In this paper, we describe the logic and theory underlying such evidence and describe traditional and modern methods for gathering and analyzing content validity data. METHOD: A comprehensive review of the literature and of the aforementioned Standards is presented. RESULTS: For educational tests and other assessments targeting knowledge and skill possessed by examinees, validity evidence based on test content is necessary for building a validity argument to support the use of a test for a particular purpose. CONCLUSIONS: By following the methods described in this article, practitioners have a wide arsenal of tools available for determining how well the content of an assessment is congruent with and appropriate for the specific testing purposes

ANTECEDENTES: la evidencia de validez basada en el contenido del test es una de las cinco formas de evidencias de validez estipuladas en los Standards for Educational and Psychological Testing de la American Educational Research Association. En este artículo describimos la lógica y teoría que subyace a tal fuente de evidencia, junto a métodos tradicionales y modernos para obtener y analizar los datos de validez de contenido. MÉTODO: una revisión comprehensiva de la bibliografía y de los mencionados Standards. RESULTADOS: para los tests educativos y otras evaluaciones de los conocimientos y habilidades que poseen los examinados, la evidencia de validez basada en el contenido del test es necesaria para elaborar un argumento de validez que apoye el uso de un test para un objetivo particular. CONCLUSIONES: siguiendo los métodos descritos en este artículo, los profesionales tienen un amplio arsenal de herramientas disponibles para determinar en qué medida el contenido de una evaluación es congruente y apropiado para los objetivos específicos de la evaluación

Validating assessments: Introduction to the Special Section.

BACKGROUND: Validation is the process of providing evidence that tests and questionnaires are adequately and appropriately fulfilling the purposes for which they are developed. In this special issue, experts from several countries describe specific approaches to test validation and provide examples of their approach. These approaches and examples illustrate the validation framework implied by the Standards for Educational and Psychological Testing. METHOD: We describe the Standards' approach for building a validity argument based on validity evidence based on test content, response processes, internal structure, relations to other variables, and testing consequences. RESULTS: The five articles provide comprehensive examples of gathering data regarding these five sources of evidence and how they contribute to the validation of the use of test scores for particular purposes. CONCLUSIONS: These five articles provide concrete examples of how the five sources of validity evidence suggested by the Standards can be used to develop a sound validity argument to support the use of a test for its intended purposes.

Validity evidence based on test content.

BACKGROUND: Validity evidence based on test content is one of the five forms of validity evidence stipulated in the Standards for Educational and Psychological Testing developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. In this paper, we describe the logic and theory underlying such evidence and describe traditional and modern methods for gathering and analyzing content validity data. METHOD: A comprehensive review of the literature and of the aforementioned Standards is presented. RESULTS: For educational tests and other assessments targeting knowledge and skill possessed by examinees, validity evidence based on test content is necessary for building a validity argument to support the use of a test for a particular purpose. CONCLUSIONS: By following the methods described in this article, practitioners have a wide arsenal of tools available for determining how well the content of an assessment is congruent with and appropriate for the specific testing purposes.